[Fix](ShortCircuit) fix prepared statement with partial arguments prepared #45371

eldenmoon · 2024-12-12T15:53:45Z

What problem does this PR solve?

We should record the placehold id map to both real Expr and the slot of conjuncts.Otherwise the info is lost, and lead to the conjuncts updated in wrong order(updateScanNodeConjuncts)
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…pared We should record the placehold id map to both real Expr and the slot of conjuncts.Otherwise the info is lost

hello-stephen · 2024-12-12T15:53:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2024-12-12T15:54:24Z

run buildall

doris-robot · 2024-12-12T16:33:59Z

TPC-H: Total hot run time: 39860 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3422a0366e5d43450f553c64976534297da5d1c, data reload: false

------ Round 1 ----------------------------------
q1	17625	7408	7264	7264
q2	2045	192	168	168
q3	10671	1103	1132	1103
q4	10535	711	726	711
q5	7588	2713	2677	2677
q6	236	149	144	144
q7	973	614	609	609
q8	9218	1869	1918	1869
q9	6710	6477	6524	6477
q10	7058	2312	2342	2312
q11	465	259	257	257
q12	429	223	221	221
q13	17773	2998	3010	2998
q14	237	212	207	207
q15	584	520	527	520
q16	653	578	590	578
q17	961	514	532	514
q18	7242	6647	6560	6560
q19	1433	1021	1069	1021
q20	465	177	176	176
q21	4202	3179	3169	3169
q22	378	324	305	305
Total cold run time: 107481 ms
Total hot run time: 39860 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7261	7253	7285	7253
q2	322	226	222	222
q3	2909	2784	2970	2784
q4	2114	1891	1833	1833
q5	5672	5652	5666	5652
q6	234	142	150	142
q7	2255	1875	1787	1787
q8	3426	3547	3489	3489
q9	8978	9055	8995	8995
q10	3594	3541	3566	3541
q11	584	507	510	507
q12	829	637	641	637
q13	9630	3240	3214	3214
q14	296	285	271	271
q15	578	547	528	528
q16	686	667	660	660
q17	1854	1656	1596	1596
q18	8310	7695	7725	7695
q19	1746	1665	1647	1647
q20	2105	1882	1881	1881
q21	5682	5431	5377	5377
q22	639	605	555	555
Total cold run time: 69704 ms
Total hot run time: 60266 ms

doris-robot · 2024-12-12T16:46:36Z

TPC-DS: Total hot run time: 197601 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3422a0366e5d43450f553c64976534297da5d1c, data reload: false

query1	1265	989	931	931
query2	6228	2141	2081	2081
query3	10949	4498	4412	4412
query4	66563	28412	23480	23480
query5	5014	486	474	474
query6	417	184	206	184
query7	5470	303	297	297
query8	345	243	245	243
query9	8397	2731	2726	2726
query10	445	249	255	249
query11	16991	15297	15772	15297
query12	152	102	106	102
query13	1446	430	429	429
query14	10576	7618	7263	7263
query15	214	184	190	184
query16	7204	476	437	437
query17	1140	563	549	549
query18	1885	313	291	291
query19	200	153	160	153
query20	128	109	113	109
query21	215	106	103	103
query22	4649	4537	4655	4537
query23	34567	34685	35038	34685
query24	5535	2487	2583	2487
query25	477	377	381	377
query26	656	148	153	148
query27	1886	283	292	283
query28	4494	2520	2487	2487
query29	657	422	439	422
query30	218	151	156	151
query31	1038	831	840	831
query32	71	51	56	51
query33	419	294	291	291
query34	954	537	505	505
query35	896	785	749	749
query36	1094	961	940	940
query37	119	72	71	71
query38	4565	4333	4471	4333
query39	1513	1447	1494	1447
query40	207	113	103	103
query41	47	47	52	47
query42	114	100	99	99
query43	555	517	511	511
query44	1243	870	846	846
query45	201	174	169	169
query46	1197	725	746	725
query47	2059	1959	1955	1955
query48	418	320	362	320
query49	729	409	379	379
query50	840	398	390	390
query51	7474	7093	7094	7093
query52	95	86	90	86
query53	256	179	176	176
query54	495	387	411	387
query55	77	78	79	78
query56	235	228	238	228
query57	1239	1122	1143	1122
query58	237	214	239	214
query59	3213	3338	3184	3184
query60	261	237	239	237
query61	104	107	101	101
query62	760	683	654	654
query63	213	183	192	183
query64	1381	655	648	648
query65	3262	3206	3234	3206
query66	624	300	294	294
query67	15917	15607	15586	15586
query68	3992	567	587	567
query69	413	249	259	249
query70	1176	1052	1205	1052
query71	379	257	254	254
query72	6062	4015	4001	4001
query73	780	356	367	356
query74	9903	8908	8913	8908
query75	3384	2661	2664	2661
query76	1963	989	1040	989
query77	516	364	318	318
query78	10434	9440	9678	9440
query79	1425	630	604	604
query80	848	475	442	442
query81	488	236	238	236
query82	1318	121	118	118
query83	165	146	143	143
query84	275	72	68	68
query85	942	299	314	299
query86	328	298	297	297
query87	4608	4491	4575	4491
query88	3551	2235	2197	2197
query89	411	302	286	286
query90	1867	185	183	183
query91	137	105	107	105
query92	63	50	51	50
query93	1976	553	562	553
query94	712	293	284	284
query95	346	246	250	246
query96	618	275	276	275
query97	2826	2664	2621	2621
query98	213	192	200	192
query99	1600	1310	1306	1306
Total cold run time: 316224 ms
Total hot run time: 197601 ms

doris-robot · 2024-12-12T16:52:06Z

ClickBench: Total hot run time: 32.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b3422a0366e5d43450f553c64976534297da5d1c, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.44	0.38	0.39
query6	1.19	0.67	0.65
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.58	0.53	0.49
query10	0.54	0.57	0.56
query11	0.15	0.11	0.11
query12	0.13	0.11	0.12
query13	0.61	0.60	0.60
query14	2.82	2.85	2.82
query15	0.91	0.83	0.82
query16	0.38	0.38	0.39
query17	1.07	0.99	1.04
query18	0.23	0.21	0.21
query19	1.88	1.83	2.00
query20	0.01	0.01	0.01
query21	15.36	0.59	0.58
query22	2.75	1.79	2.13
query23	17.02	0.83	0.83
query24	3.53	2.33	0.92
query25	0.24	0.06	0.19
query26	0.67	0.15	0.14
query27	0.04	0.05	0.04
query28	9.81	1.09	1.08
query29	12.56	3.17	3.20
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.26	0.46	0.46
query33	3.04	2.99	3.15
query34	17.17	4.42	4.46
query35	4.52	4.54	4.51
query36	0.66	0.48	0.48
query37	0.10	0.06	0.07
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.2 s
Total hot run time: 32.45 s

eldenmoon · 2024-12-13T03:20:13Z

run buildall

doris-robot · 2024-12-13T03:40:36Z

TPC-H: Total hot run time: 39839 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dbd03c904a965df6300d735d7473e209ebce9d37, data reload: false

------ Round 1 ----------------------------------
q1	17643	7458	7254	7254
q2	2050	177	167	167
q3	10624	1111	1148	1111
q4	10243	799	837	799
q5	7581	2648	2613	2613
q6	238	148	147	147
q7	983	632	619	619
q8	9241	1843	1884	1843
q9	6555	6406	6416	6406
q10	7006	2241	2280	2241
q11	472	266	264	264
q12	420	226	230	226
q13	17773	3071	3030	3030
q14	239	207	212	207
q15	565	536	537	536
q16	659	581	611	581
q17	988	570	547	547
q18	7228	6810	6664	6664
q19	1370	991	959	959
q20	464	177	180	177
q21	4016	3148	3344	3148
q22	381	300	324	300
Total cold run time: 106739 ms
Total hot run time: 39839 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7234	7252	7238	7238
q2	331	225	228	225
q3	2901	2770	2971	2770
q4	2152	1807	1792	1792
q5	5524	5648	5627	5627
q6	225	140	141	140
q7	2182	1749	1802	1749
q8	3338	3522	3462	3462
q9	9033	8977	8912	8912
q10	3587	3566	3542	3542
q11	591	510	498	498
q12	802	641	619	619
q13	11665	3292	3217	3217
q14	292	285	262	262
q15	573	524	511	511
q16	678	635	652	635
q17	1839	1582	1547	1547
q18	7795	7300	7685	7300
q19	1683	1621	1550	1550
q20	2067	1791	1867	1791
q21	5320	5261	5250	5250
q22	640	559	558	558
Total cold run time: 70452 ms
Total hot run time: 59195 ms

doris-robot · 2024-12-13T03:52:38Z

TPC-DS: Total hot run time: 190209 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dbd03c904a965df6300d735d7473e209ebce9d37, data reload: false

query1	988	383	382	382
query2	6521	2132	2065	2065
query3	6729	227	216	216
query4	33869	23435	23348	23348
query5	4320	457	449	449
query6	280	184	181	181
query7	4627	303	311	303
query8	302	231	240	231
query9	9534	2717	2718	2717
query10	480	251	264	251
query11	17887	15123	15187	15123
query12	153	108	102	102
query13	1667	433	430	430
query14	10200	7661	7573	7573
query15	323	186	181	181
query16	8145	451	467	451
query17	1807	573	569	569
query18	2155	303	301	301
query19	400	152	145	145
query20	117	112	108	108
query21	210	101	100	100
query22	4486	4357	4035	4035
query23	34426	33630	33736	33630
query24	11401	2379	2446	2379
query25	663	367	379	367
query26	1767	146	146	146
query27	2687	274	282	274
query28	7710	2437	2441	2437
query29	1026	409	412	409
query30	302	153	152	152
query31	1022	808	812	808
query32	98	57	58	57
query33	770	282	286	282
query34	1003	513	533	513
query35	894	716	731	716
query36	1081	938	938	938
query37	190	76	76	76
query38	4335	4202	4175	4175
query39	1459	1399	1403	1399
query40	284	100	101	100
query41	44	42	44	42
query42	110	100	102	100
query43	542	497	488	488
query44	1286	830	824	824
query45	188	165	171	165
query46	1166	714	705	705
query47	1974	1855	1830	1830
query48	410	314	330	314
query49	1289	409	405	405
query50	813	392	376	376
query51	7169	7023	7044	7023
query52	104	88	89	88
query53	247	179	180	179
query54	1147	398	407	398
query55	76	86	77	77
query56	254	236	225	225
query57	1272	1079	1111	1079
query58	229	219	209	209
query59	3295	3060	3068	3060
query60	266	245	240	240
query61	116	108	121	108
query62	930	678	673	673
query63	215	188	189	188
query64	4995	692	677	677
query65	3290	3172	3235	3172
query66	1444	316	308	308
query67	15803	15500	15461	15461
query68	5010	570	545	545
query69	412	247	265	247
query70	1186	1193	1135	1135
query71	319	251	255	251
query72	6322	4027	4045	4027
query73	759	358	363	358
query74	10342	8803	8857	8803
query75	3445	2621	2685	2621
query76	3053	1017	1011	1011
query77	537	265	302	265
query78	10330	9412	9416	9412
query79	1775	604	595	595
query80	1135	444	424	424
query81	523	244	237	237
query82	916	120	117	117
query83	247	144	143	143
query84	239	71	69	69
query85	1384	383	303	303
query86	444	319	310	310
query87	4725	4668	4447	4447
query88	3336	2249	2195	2195
query89	412	297	289	289
query90	2148	189	191	189
query91	151	106	106	106
query92	66	50	51	50
query93	1591	550	553	550
query94	1096	296	294	294
query95	373	250	248	248
query96	616	284	283	283
query97	2781	2631	2658	2631
query98	215	198	203	198
query99	1573	1337	1323	1323
Total cold run time: 302453 ms
Total hot run time: 190209 ms

doris-robot · 2024-12-13T03:58:09Z

ClickBench: Total hot run time: 32.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dbd03c904a965df6300d735d7473e209ebce9d37, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.61	0.10	0.10
query5	0.44	0.40	0.41
query6	1.14	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.52	0.49
query10	0.56	0.58	0.57
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.62	0.60	0.61
query14	2.82	2.74	2.81
query15	0.89	0.82	0.82
query16	0.38	0.37	0.38
query17	1.07	1.03	1.07
query18	0.22	0.21	0.23
query19	1.96	1.83	2.03
query20	0.02	0.01	0.01
query21	15.36	0.61	0.60
query22	2.80	3.01	1.46
query23	17.04	1.01	0.92
query24	3.37	1.22	1.30
query25	0.24	0.19	0.12
query26	0.44	0.14	0.13
query27	0.04	0.05	0.05
query28	10.41	1.10	1.07
query29	12.56	3.25	3.21
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.32	0.45	0.45
query33	3.03	3.12	2.99
query34	17.07	4.40	4.49
query35	4.47	4.47	4.43
query36	0.66	0.47	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.03
query40	0.19	0.12	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.39 s
Total hot run time: 32.5 s

morrySnow · 2024-12-13T04:05:37Z

regression-test/suites/point_query_p0/test_point_query.groovy

+    """
+    sql "insert into test_partial_prepared_statement values ('user_guid', 'feature', 'sk','feature_value', '2021-01-01 00:00:00')"
+    def result2 = connect(user, password, prepare_url) {
+        def partial_prepared_stmt = prepareStatement "select /*+ SET_VAR(enable_nereids_planner=true) */ * from regression_test_point_query_p0.test_partial_prepared_statement where sk = 'sk' and user_guid = 'user_guid' and  feature = ? "


enable_nereids_planner is useless, could be removed

github-actions · 2024-12-13T04:14:43Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-13T04:14:45Z

PR approved by anyone and no changes requested.

eldenmoon · 2024-12-13T06:16:37Z

run buildall

github-actions · 2024-12-16T03:25:15Z

PR approved by at least one committer and no changes requested.

csun5285

LGTM

…pared (#45371) We should record the placehold id map to both real Expr and the slot of conjuncts.Otherwise the info is lost, and lead to the conjuncts updated in wrong order(`updateScanNodeConjuncts`)

…rguments prepared #45371 (#45465) Cherry-picked from #45371 Co-authored-by: lihangyu <[email protected]>

…pared (apache#45371) We should record the placehold id map to both real Expr and the slot of conjuncts.Otherwise the info is lost, and lead to the conjuncts updated in wrong order(`updateScanNodeConjuncts`)

…pared (#45371) (#45732) (#45371)

eldenmoon added 2 commits December 12, 2024 23:51

add case

d4adf92

[Fix](ShortCircuit) fix prepared statement with partial arguments pre…

b3422a0

…pared We should record the placehold id map to both real Expr and the slot of conjuncts.Otherwise the info is lost

eldenmoon added dev/2.1.x dev/3.0.x labels Dec 12, 2024

fix

dbd03c9

eldenmoon closed this Dec 13, 2024

eldenmoon reopened this Dec 13, 2024

morrySnow reviewed Dec 13, 2024

View reviewed changes

morrySnow requested a review from 924060929 December 13, 2024 04:06

eldenmoon added the usercase Important user case type label label Dec 13, 2024

924060929 previously approved these changes Dec 13, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 13, 2024

github-actions bot added the reviewed label Dec 13, 2024

fix ut

baaacd0

eldenmoon dismissed 924060929’s stale review via baaacd0 December 13, 2024 06:16

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Dec 13, 2024

qidaye approved these changes Dec 16, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 16, 2024

csun5285 approved these changes Dec 16, 2024

View reviewed changes

eldenmoon merged commit 91c475e into apache:master Dec 16, 2024
27 of 29 checks passed

eldenmoon deleted the fix-short-circuit-prep-order branch December 16, 2024 07:49

github-actions bot mentioned this pull request Dec 16, 2024

branch-3.0: [Fix](ShortCircuit) fix prepared statement with partial arguments prepared #45371 #45465

Merged

github-actions bot added the dev/2.1.x-conflict label Dec 16, 2024

eldenmoon added a commit that referenced this pull request Dec 17, 2024

branch-3.0: [Fix](ShortCircuit) fix prepared statement with partial a…

820b300

…rguments prepared #45371 (#45465) Cherry-picked from #45371 Co-authored-by: lihangyu <[email protected]>

eldenmoon added dev/3.0.4-merged and removed dev/3.0.x labels Dec 17, 2024

eldenmoon mentioned this pull request Dec 20, 2024

[Fix](ShortCircuit) fix prepared statement with partial arguments prepared (#45371) #45732

Merged

yiguolei pushed a commit that referenced this pull request Dec 23, 2024

[Fix](ShortCircuit) fix prepared statement with partial arguments pre…

ca9f0ad

…pared (#45371) (#45732) (#45371)

yiguolei added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Dec 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix](ShortCircuit) fix prepared statement with partial arguments prepared #45371

[Fix](ShortCircuit) fix prepared statement with partial arguments prepared #45371

eldenmoon commented Dec 12, 2024 •

edited

Loading

hello-stephen commented Dec 12, 2024

eldenmoon commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

eldenmoon commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

morrySnow Dec 13, 2024

eldenmoon Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

eldenmoon commented Dec 13, 2024

github-actions bot commented Dec 16, 2024

csun5285 left a comment

[Fix](ShortCircuit) fix prepared statement with partial arguments prepared #45371

[Fix](ShortCircuit) fix prepared statement with partial arguments prepared #45371

Conversation

eldenmoon commented Dec 12, 2024 • edited Loading

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

hello-stephen commented Dec 12, 2024

eldenmoon commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

doris-robot commented Dec 12, 2024

eldenmoon commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

doris-robot commented Dec 13, 2024

morrySnow Dec 13, 2024

Choose a reason for hiding this comment

eldenmoon Dec 13, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

eldenmoon commented Dec 13, 2024

github-actions bot commented Dec 16, 2024

csun5285 left a comment

Choose a reason for hiding this comment

eldenmoon commented Dec 12, 2024 •

edited

Loading